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Abstract 

We use power sums plethysm operators to introduce H functions which interpolate between 
the Weyl characters and the Hall-Littlewood functions Q' corresponding to classical Lie groups. 
The coefficients of these functions on the basis of Weyl characters are parabolic Kazhdan-Lusztig 
polynomials and thus, by works of Kashiwara and Tanisaki, are nonnegative. We prove that they can 
be regarded as quantizations of branching coefficients obtained by restriction to certain subgroups 
of Levi type. The H functions associated to linear groups coincide with the polynomials introduced 
by Lascoux Leclerc and Thibon in [J. Math. Phys 38 (1996), 1041-1068]. 

1 Introduction 

Given [i a partition with at most n parts, the Hall Littlewood function Q'^ can be defined by 

A 

where the sum runs over the partitions of length at most n, K\^{q) is the Lusztig (/-analogue of weight 
multiplicity associated with (A,//) and s\ the Schur function indexed by A, that is the Weyl character 
of the irreducible finite dimensional GL n -module V(A). Since K\ „(l) is equal to the dimension of the 
weight space \i in V(A), Q'^ can be regarded as a quantization of the homogeneous function hp. In [8], 
Lascoux, Leclerc and Thibon have introduced a new family of symmetric functions £Z~ depending on a 
fixed nonnegative integer £ which interpolate between the Schur functions and the Hall-Littlewood 
functions Q' . The polynomials Hi can be combinatorially described in terms of the spin statistic on 
certain generalized Young tableaux called ^-ribbon tableaux. These ribbon tableaux naturally appear 
in the description of the action of the power sum plethysm ipe on symmetric functions. Recall that for 
any symmetric function /, ifig(f) is obtained by replacing in / each variable x% by xf. In particular 
ij)£ multiplies the degrees by t. The space of symmetric functions is endowed with an inner product 
< •, • > which makes the basis of Schur functions orthonormal. Then <p^, the adjoint operator of 
if)£, divides the degrees by £. It is well known that ip^Sn) can be computed from the Jacobi-Trudi 
determinantal identity. Namely, we have 

= or <Pi( s ») = e (°"o)v°) ' ' ' S »V-V (!) 

where e(<7n) = il is the signature of a permutation a$ G S n and (fi^ , ^~^) a ^-tuple of partitions 
defined by i and fi. By expanding ^(s«) on the basis of Schur functions we obtain then 

<PtM = ^o) C M0),...,^-D s A (2) 
A 
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where c^ (0) is the Littlewood-Richardson coefficient giving the multiplicity of V(A) in the tensor 

product V(ji<®) g> • • • ® V^" 1 ). When £ = 1, one has (pg(sg^ — and when t ~> n one can prove 
that tpi(s£n) = h^. Thus the functions lijp = e(o"o)^(s^) interpolate between the functions s M and 
/i„ and have nonnegative coefficients on the basis of Schur functions. 

In |8j , the authors have interpreted the algebra of symmetric functions as the bosonic Fock space rep- 
resentation of the quantum affine Lie algebra U q (sl n ). This permits to introduce a natural quantization 
tp qt £ of the power sum plethysm Let (p q£ be the adjoint operator of ip„£ with respect to < •, • >. 
The function i?~ is then defined as a simple renormalization of fq^{si^). This gives an identity of the 
form 

H i = X) c i(°),...,^- 1 >^ SA 

A 

where the polynomial c^ {0) ^(t-i) (<?) is a ^-analogue of c^ {0) ^y-iy 

Lusztig's g-analogues K\^(q) are particular affine Kazhdan-Lusztig polynomials. These polynomials 
arise in affine Hecke algebra theory as the entries of the transition matrix between the natural basis 
and a special basis defined by Lusztig. By replacing the affine Hecke algebra H by one of its parabolic 
modules Hu {y being a weight of the affine root system under consideration), Deodhar has introduced 
analogues of the Kazhdan-Lusztig polynomials. In |9j, it is shown that the family constituted by 
these parabolic Kazhdan-Lusztig polynomials contains in particular the g-analogues c A (0) a-i)(o)- 
By a result of Kashiwara and Tanisaki [7], this implies notably that the coefficients of the polynomial 
c^ (0) (q) are nonnegative integers. 

The aim of the paper is to introduce analogues of the polynomials for the classical Lie groups 
G = S02n+i, Sp2n and SC-2n which interpolate between the Weyl characters and the Hall-Littlewood 
functions associated with G. Write also s\ for the Weyl character of the irreducible G-module V(A) 
of highest weight A. We define the plethysm operator tp£ and its dual ^ on the Z-algebra generated 
by these Weyl characters. By a subgroup L C G of Levi type, we mean a subgroup of G isomorphic 
to the Levi subgroup of one of its parabolic subgroups. Given 7 a highest weight of L, we denote by 
[V(A) : Vl(7)] the multiplicity of the irreducible L- module Vl(j) of highest weight 7 in the restriction 
of V(A) to L. Then, provided that £ is odd when G = Sp2 n or S02 n , we establish for any Weyl 
character s M such that ^(s^) 7^ 0, a formula of the type 

Ms») = e(w )^[V(\) :V L (fy]s x (3) 

where e(wo) is the signature of an element wo £ W the Weyl group of G, L a subgroup of Levi 
type of G and Of) a dominant weight associated with L. The procedure which yields wq,L and 
(fy from i and fi can be regarded as an analogue of the algorithm computing the ^-quotient of a 
partition which implicitly appears in ([T]). The identity ([2]) can also be rewritten as in ([3]). Indeed, take 
L = GL ro x---xGL rei where for any k = 1, ...,£— 1, r k is the length of Then (^) = (fJ,(°\ /i^" 1 )) 
can be interpreted as a dominant weight for the subgroup of Levi type L of GL n and we have the 
duality c£ C0)j ^- 1} = [V(\):V L (»)}. 

The surprising constraint £ odd when G = Sp2 n or S02 n appearing in (|3|) follows from the fact 
that the procedure giving wq^L and Of) mentioned above depends not only on the Lie group under 
consideration but also on the parity of the integer I. For G = S02n+i the coefficients of e{wo)ip ^(s ff) 
on the basis of Weyl characters are always branching coefficients corresponding to restriction to L. 
For G = Sp2n or S02 n this is only true when i is odd. Note that this difficulty disappears in large 
rank, that is for n> £\fj,\ (but see Section [6.4p . 
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To define the functions H in type B, C or D, we prove the equalities 

|< V^(sa),^ >| = |< sx^eis^) >| = P~ +p>a+p (l) 

which show that the coefficients of the expansion of ifjg(s\) on the basis of Weyl characters are, up to 
a sign, parabolic Kazhdan-Lusztig polynomials specialized at q = 1. By using (J3]) this gives, providing 
i is odd for G = Sp2 n or S02 n 

We then introduce the functions 

where [V(X) : Vl(^)] 9 = P^ +p ix+pil)- This yields nonnegative g-analogues of the branching coefficients 
[V(\) : Vl(i)]- The functions H p are then defined by setting H p = G\ . We obtain the identities 
= s p and H p = Q'^ when £ is sufficiently large. Thus the functions H p interpolate between the 
Weyl characters and the Hall-Littlewood functions associated with G. 



The paper is organized as follows. In Section 2 we recall the necessary background on classical root 
systems, Weyl characters, subgroups of Levi type and their corresponding branching coefficients. In 
Section 3, we define the plethysm operators ipg and their dual operators (f£. By abuse of notation, we 
also denote by ipg the linear operator on the group algebra Z[Z n ] with basis the formal exponentials 
(e' 3 ) such that 

¥>M e ) = S „ • for any 3 G Z". 

' \ otherwise 

We then show how the determination of <£i{s p ) can be reduced to the computation of the polynomial 

n ( i - e °)) 

a£R+ 

where -R+ is the set of positive roots corresponding to the Lie group G. This permits to establish 
formulas ([3]) providing £ is odd when G = Sp2 n or SC^n- For completion we have also included the 
case G = GL n and shown why (|3|) cannot hold when I is even and G = Sp2 n or SC^n- To make the 
paper self-contained, we have summarized in Section 4 some necessary results on affine Hecke algebras 
and parabolic Kazhdan-Lusztig polynomials. Section 5 is devoted to the definition of the polynomials 
G p and H p and to their links with the Weyl characters and the Hall-Littlewood functions. Finally 
we briefly discuss in Section 6 the problem of defining nonnegative (/-analogues of tensor product 
multiplicities when G ^ GL n . We add also a few remarks concerning the exceptional root systems. 

Acknowledgments: The author wants to thank B. Leclerc for very helpful and stimulating discus- 
sions on the results of [8] and [9]. 
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2 Background 



2.1 Classical root systems 

In the sequel G is one of the complex Lie groups GL n , Sp2n, SC>2n+i or S02n and g its Lie algebra. 
We follow the convention of [7\ to realize G as a subgroup of GL^ and q as a subalgebra of gl N where 



N 



n when G = GL n 

2n when G = Sp2 n or S02 n 

2n + 1 when G = S02n+i 



With this convention the maximal torus T of G and the Cartan subalgebra f) of q coincide respectively 
with the subgroup and the subalgebra of diagonal matrices of G and q. Similarly the Borel subgroup 
B of G and the Borel subalgebra b of q coincide respectively with the subgroup and subalgebra of 
upper triangular matrices of G and g. 

Let djy be the linear subspace of gl N consisting of the diagonal matrices. For any i 6 I n = {1, ■■■,n}, 
write Si for the linear map e, : dpj — > C such that £i(D) = 8 n -i+% for any diagonal matrix D whose 
(z, i)-coefficient is 5{. Then (ex, •••,e n ) is an orthonormal basis of the Euclidean space fjj (the real part 
of fj*). Let (•, •) be the corresponding nondegenerate symmetric bilinear form defined on t)t. Write R 
for the root system associated with G. For any a £ R we set a v = . The Lie algebra q admits 
the diagonal decomposition fl = fj ® ® agfl g Q . We take for the set of positive roots: 

R + = {sj — £i with 1 < i < j < n} for the root system A n -\ 

R + = {ej — £i,£j + £i with 1 < i < j < n} U {£{ with 1 < i < n} for the root system B n 
R + = {sj — £{,£j + £i with 1 < i < j < n} U {2ej with 1 < i < n} for the root system C n 
R + = {ej — £i,£j + £i with 1 < i < j < n} for the root system D n 

Let p be the half sum of positive roots. Set J n = {n < ■ ■ ■ < 1 < 1 < • • • < n} where, for each integer 
i = l,...,n, we have written % for the negative integer —%. For any x £ J n we have x = x and we 
set \x\ = x if x > 0, \x\ = x otherwise. Given a subset U C J n , we define \U\ = {\x\ \ x € U} and 
U = {x | x e U}. 

The Weyl group of GL n is the symmetric group S n and for G = SO211+1, Sp2n or S02n, the Weyl group 
W of the Lie group G is the subgroup of the permutation group of J n generated by the permutations 

Si = + 1)(m + 1)) * = 1) •••! n — 1 arm s n = ( n i^) f° r tli e root systems i3 n and C n 
Si = + l)(i, i + 1), i = 1, n — 1 and = (n, n — l)(n — 1, n) for the root system D n 

where for a 7^ b (a, b) is the simple transposition which switches a and b. For types -B n and C n , W is 
the group of signed permutations. It is the subgroup of the permutation group of J n consisting of the 
permutations w such that w(i) = w(i). For type D n , the elements of W verify the additional constraint 
cardji € I n \ w{i) < 0} € 2N. We identify the subgroup of W generated by Sj = (i, i + + 1), 

i = 1, n — 1 with the symmetric group S n . The signature e of to € W is defined by e(u;) = (— l)'^) 
where I is the length function corresponding to the above sets of generators. Consider the increasing 
sequence K = (i p , ~.,i p ) C J n . For X = B, D set 

Wx,k = {w £ W of type X n | u;(x) = x for any x ^ i^T}. 

Then, Wx,k is isomorphic to the Weyl group of type X p . Let £x,K be the corresponding signature. 

Lemma 2.1.1 Consider X = B,D and w G Wx,k- Then we have £x,k{w) = £(w). 
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Proof. Suppose X = B. The generators of the Weyl group Wx : k are the tk = (ik, ik+i)(ik, *fc+i)> 
fc = 1, ...,p — 1 and s n = (ip,i p ). One verifies easily that, considered as elements of W, they have an 
odd length. We proceed similarly when X = D. m 

The action of w G W on (3 = ((3 1 , (3 n ) G f)^ is defined by 

w((3 1 ,...,(3 n ) = (pr~\..,W~ 1 ) (4) 

where (3f = f3 w ^ \iw(i) G and (3f = —P w (^ otherwise. The dot action of W on j3 = (f3 1 , (3 n ) G f)j|j 
is defined by 

w o [3 = w ■ ((3 + p) — p. (5) 
The fundamental weights of belong to (§)". More precisely we have cjj = (0*, 1*) G N n for z < n — 1 

and also i = n - 1 for q ^ so 2n , ^n n = (l n )i w n n = u n n = (§") and w n-i = Tne weight 

lattice P of can be considered as the Z-sublattice of (§)" generated by the cjj, i & I. For any 

/? = (3 n ) G P, we set \[3\ = (3 l -\ h /3 n . Write P + for the cone of dominant weights of G. With 

our convention, a partition of length m is a weakly increasing sequence of m nonnegative integers. 
Denote by P n the set of partitions with at most n parts. Each partition A = (Ai, A m ) G V n will 
be identified with the dominant weight YliLi ^i £ i- Then the irreducible finite dimensional polynomial 
representations of G are parametrized by the partitions of V n . For any A G V n , denote by V(A) the 
irreducible finite dimensional representation of G of highest weight A. We will also need the irreducible 
rational representations of GL n . They are indexed by the n-tuples 

(7~,7 + ) = (-77'-»-7i,7i",72>-.7^) ( 6 ) 

where 7 + = (7^, 7^, 7+) and 7" = (77, ...,7^) are partitions of length p and q such that p + q = n. 
Write V n for the set of such n-tuples and ^denote also by V(j) the irreducible rational representation 
of GL n of highest weight 7 = (7", 7 + ) G V n . 

In the sequel, our computations will also make appear root subsystems of the root systems R described 
above. Suppose that G is of type X n with X n = A n -i, B n , C n or D n . Let I = i r ) be an increasing 

sequence of integers belonging to I n , that is ik G I n for any k = 1, .., r and i± < ■ ■ ■ < i r . Then 

Ri = {a G En ® i€l Zei} 

is a root subsystem of R of type X r . Write Rf for the set of positive roots in Rj. Then we have 
Rf = Rjf\ R + . The dominant weights associated with Rj have the form A = (Ai, A n ) where A« / 
only if i G / and A* 7 -* = (A^, A« r ) G V r . We slightly abuse the notation by identifying A with A*- 1 - 1 . 
Consider an increasing sequence X = (x±, ...,x r ) of integers belonging to J n such that \x^\ = \xk'\ if 
and only if k = k'. For any integer i = 1, n, set £j = —8%. Then 

Ra,x = {M £ xj ~ e Xi ) I 1 < * < 3 < r ) 

is a root subsystem of R of type A r -i. To see this, consider the linear map Ox '■ Z r — > Z ra such that 
0x{ £ i) = £x t - The map is injective and preserves the scalar product in 27 and Z™ . Moreover the root 
system — £?) | 1 < « < J < r} C HI of type ^4 r is sent on Ra,x by The set of positive roots 

in Ra,x is equal to x = Ra,x H P + . Denote by s G {1, r} the maximal integer such that x s < 0. 
We associate to X, the increasing sequence of indices I C I n defined by 
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It will be useful to consider the weights corresponding to Ra,x as the r-tuples (3 = (@ Xl , ...,/3 Xr ) with 
coordinates indexed by X. The coordinates (01, j3' n ) of (5 on the initial basis (ei,...,e n ) are such 
that (3\ = (5 Xa if i = x a G X, { = —(3 Xa if i = x a G X and { = otherwise. With this convention the 
dominant weights for Ra,x have the form 



This simply means that we have chosen to expand the weights of Ra,x on the basis {e x \ x G X} 
rather than on the basis {ei \ i G /} to preserve the identification of the dominant weights with the 
nondecreasing r-tuples of integers. 

Example 2.1.2 Take G = Sp w . 

• For I = (2, 4, 5) we have 



which is the set of positive roots of a root system of type C3. The weight A = (1, 2, 2) is dominant 
for Gi. Considered as a weight of Spio, we have A = (0, 1, 0, 2, 2). 

• For X = (5, 2,1,4) we have 



which is the set of positive roots of a root system of type A3. The weight 7 = (—3, —1,4,5) is 
dominant for Gx- Considered as a weight of Spio, we have 7 = (4,1,0,5,3). 

2.2 Subgroups of Levi type 

Suppose G is a classical Lie group and consider R the corresponding root system. We shall need Lie 
subgroups of G associated with particular sub-root systems of R. Each of these subgroups will be of 
Levy type, that is, will be isomorphic to the Levi subgroup of one of the parabolic subgroups of G. 
Consider p > 1 an integer. Let 1^ = •••) 4?) De an increasing sequence of integers in I n . For 
k = l,...,p, consider increasing sequences 

X (k) = (^...^({O) c j n suc h that card(XW) = r k . Let 
5^ be maximal in {1, ...,r k } such that x$ < 0. Set 



AW = (A X1 , X Xr ) G V r - 



(8) 



Rj = {e 5 ± £4, £5 ± e 2 , £4 ± £2, 2£2, 2£ 4 , 2£ 5 } 



R~A,X — { £ 4 — £l) £5 — ^2, £l + £2,^1 + ^5) £4 + ^2, £4 + £5} 



/(*) 




X l ) X s fc +1' X r k 




(9) 



We suppose that the sets I^ k \ k 
{m,XW,...,XW} and 



0, ...,p are pairwise disjoint and verify u| =0 /( fe ) = I n . Set X 




fc=i 



Then Qj = f) © QgRl Q is a Lie subalgebra of g. Its corresponding Lie group G% is a subgroup of G 
of Levi type and we have 



x • • • x GL rp for G = GL n 
SO 2r0+1 x GL ri x • • • x GL rp for G = S0 2n +i 
Sp2r x GL ri x • • • x GL rp for G = Sp 2n 
SC>2r x GL ri x • • • x GL r . p for G = S , 02 ri 
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The root system associated with Gj is Rj- Denote by Pj its cone of dominant weights. The weight 
lattice of G% coincides with that of G since the Lie algebras qj and q have the same Cartan subalgebra. 
The elements of P+ are the (p + l)-tuples A = (A (0) , A (1) , A (p) ) where A (0) = (A* | i £ I (0) ) is a 
dominant weight of Rq j(o) and for any k = l,...,p, X^ k ' = (Aj [ i £ is a dominant weight of 

R G X (k). For any A £ P^" , we denote by Vx(A) the irreducible finite dimensional Gj-module of highest 
weight A. Each weig can be considered as a weight /? = ■■■,P' n ) of 

P. With the convention ([8]) we have then P\ = /3.(o) if i = € I*- ** and for any k = 1, ...,p, p\ = P.(k) 

if i = E X^), f3[ = — /3.(fe) if i = E X^. In the sequel we identify the two expressions 

la 

(3 = G0<°>,/?W and /? = (/?;, ...,#J (10) 

of the weights of Pj. 



2.3 Weyl characters and dual bases 

We refer the reader to [13] and [15] for a detailed exposition of the results used in this paragraph. We 
use as a basis of the group algebra Z[Z n ], the formal exponentials (e^)p & z n satisfying the relations 
e /3i e /3 2 _ e /3i+/32. "We furthermore introduce n independent indeterminates xi,...,x n in order to identify 
Z[Z n ] with the ring of polynomials Z[xi, x n , x^ 1 , ..^x^ 1 ] by writing e' 3 = x-y 1 ■ ■ ■ Xn n = x@ for any 
P = (/?!, ...,P n ) £ 1 n . Define the action of the Weyl group W on Z[Z n ] by w ■ x? = x w ^ . The Weyl 
character sp is defined by 



S 



p where ap = e(a)(w ■ x 13 ) 



For any p £ Z n we have 



J e(w)sA if there exists w £ W and A £ P ra such that A = it; o /3 . . 

S ^ 1 otherwise 

Let A be the Z-algebra generated by the characters s\,\ £ P ra . For any /3 £ Z n , denote by the 
stabilizer of /? under the action of the Weyl group W and by W 13 a set of representatives in W/Wp 
with minimal length. Then the functions 



w ■ x 13 



belong to A. Moreover {m\ | A £ V n } is a basis of A. We have the decomposition 

s\ = Kx^rrhfi (12) 

where -KTa,^ is equal to the dimension of the weight space n in the irreducible representation V^(A). 
There exists an inner product < •, • > on A which makes the characters s\ orthonormal. We denote 
by {hf! | [J, £ V n } the dual basis of {m\ | A £ V n } with respect to < •, • >. The homogeneous functions 
hn are given in terms of the Weyl characters by the decomposition 

h a =J2 K x , u s x . (13) 

This decomposition is infinite in general when G ^ GL n . Nevertheless, by embedding A in the ring 
A of universal characters defined by Koike and Terada [7], it makes sense to consider formal series 
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in the characters s\, A G V n . Note that the function h p is not the character of the representation 
V(niUJ%) <g) • • • (g) V(/i n k>i) when G 7^ GL n , . For any /3 G Z n we define the function /i^ by 

= /»„ (14) 
where /i is the unique dominant weight contained in the orbit W ■ (3. 

2.4 Jacobi-Trudi identities 

Denote by C n = IK [[a;/ 3 ]] the vector space of formal series in the monomials x@ with f3 G Z. We identify 
the ring of polynomials T n = with the subspace of C n containing the finite formal series. The 

vector space C n is not a ring since (3 G Z. More precisely, the product F\ ■ ■ ■ F r of the formal series 
Fj = X/fles- * = 1' r ^ s defined if and only if, for any 7 G Z n , the number 7V 7 of decompositions 
7 = ft 1 ' + • • • + ft r ' such that ft*' G is finite and in this case we have 



F 1 ■ ■ ■ F r = NyX 7 - 



In particular the product P ■ F with P G V n and F G C n is well defined. 
Set 

v = n a^) andA= n (i-o- 

Then A E ,F n and V G £ n . We define two linear maps 

L ^^ A and H : | £ ™ ~* , A . 

X I ^ S I X> I— > Alg 

From Theorem 2.14 of [15] we obtain 

Proposition 2.4.1 For any (3 G Z n , = ^ w6H / £{w)hp +p - w . p . 
By using the identity 

A = x p Y e{w)x~ w -P (15) 
w&w 

the previous proposition is equivalent to the following identity: 

S(x /3 ) = H(A x x 13 ). (16) 

Proposition 2.4.2 For any (3 G Z n we have H(x /3 ) = S(V x x 13 ). 

Proof. Denote by xa an d Xv the linear maps defined on C n by setting Xa( x ^) = A x and 
Xv( x/3 ) = Vx/ respectively. By (fT6|) we have S = Ho^ A . Moreover for any /3 G Z n , Xa°Xv( x ^) = 
This gives S(V x x 13 ) = S o Xv( x/3 ) = H Xa Xv( x/3 ) = H(x /3 ). ■ 
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2.5 Branching coefficients for the restriction to subgroups of Levi type 

Consider X = {Io, Xi, X p } as in 12.21 The set I characterizes a subgroup Gj C G of Levi type. Set 

Ax= II (l-*")andVi= II 77^) 

aGi?+ aG-R+-i?+ 

Then A T G and Vj G £ n . Note that Vj = V x Aj. 
As a formal series, Vj can be expanded on the form 

Vx = Yl V x {l)x\ (17) 

7GZ n 

Consider A G V n and = fJr^') a dominant weight associated with G%. We denote by [V(A) : 

Vj(fi)] the multiplicity of the irreducible representation Vj(fj.) in the restriction of V(A) from G to G%. 
The proposition below follows from Theorem 8.2.1 in PQ: 

Proposition 2.5.1 Consider A G "P^ and /i = (/i^ ^, />)) a dominant weight of Pj . Then 

[V(X) : Vx{p)] = <w)Vi{w o A - /i). 

Define the linear map 

J Si : £ n ^ 2 

\ ■-> H(A X x a^) ' 

For any dominant weight ^ G -fj", set 

S^j = H(A X x i") = Sx(^). (18) 
Proposition 2.5.2 With the above notations we have 

S^j = Y i y W ■ Vr(n)]s x . (19) 

Proof. For any (3 G Z n , we have obtained in the proof of Proposition 12.4.2], the identity H(x^) = 
S o Xv( x ^)- Denote by Xa,i the linear map defined on C n by setting Xa,x( x ^) = ^? x x ^ ■ We obtain 
Sj(x /3 ) = H(Aj x x 13 ) = S o Xv o Xa,i( x(3 ) = S ( v i x x /3 )since V j = V x A x . Thus by dTTJ) this yields 
Si(x^) = X^gz™ '^ , x(7) s /3+7- F° r an Y 7) we know by (fTTj) that sg +7 = or there exists A G "P n , w G W 
such that A = u; o (/? + 7) and sp +1 = e{w)s\. This permits to write 

Si^) = y, E £ 0)?m™ ° A - /3) s a- 

When /3 = /i is a dominant weight of Pj", we obtain the desired identity by using Proposition ^. 5.11 ■ 
Remarks: 

(i) : When G = Gj that is, when = n and r± = ■ ■ ■ = r p = 0, we have fi = Aj = A and 
Hi = H. Thus = s ^(o)- This can be recovered by using (fT9|) since in this case [V(A) : Vj(fi)] = 
except when A = 
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(ii) : When Gj = H the maximal torus of G, that is when n = p + 1 and rj~ = 1 for any k = 0, 

we have ^ = fj,^~^ for any i = 1, n, Aj = 1 and Hj(x /3 ) = hp for any /? G Z n . Hence 5^ = /i^. In 

this case [V(X) : Vr(fi)] = K x ^ for any AeP n . Thus CE|) reduces to (fT5j) . 

(hi) : By (i) and (ii) the functions S^j interpolates between the Weyl characters and the homoge- 
neous functions h^. 

(iv) : When G = GL n , we have the duality 

[V(X) : Vr(jM)] = cJ C o) r .^ (p) 

where c^ {0) (p) is the Littlewood-Richardson coefficient associated with the multiplicity of V(X) in 
the tensor product = V(/j,( ') <8> • • • <8> V{^>). Thus we can write S^x = 2^Ae-p„ c^ (0) ( P ) S A- This 
means that S^j is the character of V^. Such a duality does not exist for G = Sp2 n , S02n+i or S02 n , 
(but see Section [6]). 



3 Plethysm on Weyl characters 

3.1 The operators ty e and ip £ 

Consider t a positive integer. The power sum plethysm operator ^ is defined on A be setting \P i(ms) = 
rriip for any (3 = (Pi, ...,/3 n ) G Z n where £/3 = ...,£/3 n ). Since {m^ | A G "P n } and {/ia | A € "P n } 
are dual bases for the inner product < •, • >, the adjoint operator tp^ of ^ £ verifies 

where 0/£ = (fiji, ...,P n /£) when (3 G (£Z) n . 

By abuse of notation, we also denote by *S>i and (p e the linear operators respectively defined on C n by 
setting 

Vitf) = ^ ^d Vt (xP) = { f / t h i erwise ( ^ r for ^ G Z " (21) 

Remark: Since ^(x^ 3 x x@ ) = ^fi(x^) x ^^(x^ ) for any /3, G Z ra the map is a morphism of 
algebra. This is not true for ipg. Nevertheless, if {i±,...,i r } and {ji, j s } are disjoint subsets of I n , 
i = (ti, i r ) G 27 and 7 = (7^ 7 S ) G 7L S we have 

^« • • • < r r x 4 1 • • • x lP = ■ ■ ■ <0 x ^( x Ji • • • x ]0- ( 22 ) 

For any A G V n , ^>i(s\) belongs to A, thus decomposes on the basis {s^ \ fi G V n }. Let us write 

Since and <pt are dual operators with respect to the scalar product < •, • >, we can write 
n\ tfl =< ^e(s\),s fl >=< s x , <M s m) > • So we have 

\ev n 

By (|16p and Proposition 12.4.1] we obtain the identity 

Sfj, = ^2 e{w)h^ +p ^ w . p = H(A x a^). 

Thus from ([20]) and (f2"T|) we derive v^( s m) = H(^(A x x M )). Set = A x x^. From the previous 
arguments the coefficients are determined by the computation of ipi(P^). 
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3.2 Computation of </^(P M ) 

For any i G {n, . . . , 1 } we set x\ = ^ . This permits to consider also variables indexed by negative 

i 

integers. Given X = ...,i r ) an increasing sequence contained in J n and (3 = (Pi, ...,/3 r ) G 27, we 
set x x = x^ 1 • ■ ■ x? r . We also denote by Sx the group of permutations of the set X. Each a G Sx 
determines a unique permutation a* of the set {1, ...,r} defined by 

a(i p ) = v* (p ) for any p = 1, ...r. (23) 

In the sequel, we identify for short a and a*. Similarly, given Z = (u r , ...,Tii,u\, ...,u r ) an increasing 
sequence such that {u\, ...,u r } C I n , each signed permutation w defined on Z will be identified with 
the signed permutation w* defined on J r by w(u p ) = u w *^ for any p = 1, ...r. 
Set /? n = (l,2,...,n). For any w € W we have w ■ p n = (w(l), ...,w(n)). This permits to write 

e{w)x- w - p « = e{w)x~ w{l) ■ ■ ■ x~ w(n \ (24) 



3.2.1 For G = GL n 

Set K n = (1,...,1) G Z n . Since cr(K n ) = K n for any <r G ,S n , one can replace p by p n = (l,2,...,n) in 
(|15p . By using (|24p we can write 

P M = x^ tl+1) • • • x ( n ^ +n) e(a)x- a{1) ■ ■ ■ x" CT(n) 

cr£S„ 

where p = (pi, ...,p n ). For any k G {0, ...,£ — 1} consider the ordering sequences 

= (iel n \fii + i = kmodt) and J (fc) = (i G J n | i = kmodt). (25) 
Set r k = csxd(I^) and write 1^ = (4 , )■ Then 

/,« = + Z + | i G I«) G 

We derive 

-P,Lt = x j%) x j(i) ' ' ' x j(e-i) ^ 1 £{g) x \ ' ' X n X J^J J^J x -(k) 

aeS„ fc=0a=l 

This gives 

<piW = 44J ■ • • 4-1 E f n ft ^ fc)) -H • (26) 

o-eS„ \k=0a=l a ) 

The contribution of a fixed permutation <r G S n in the above sum is nonzero if and only if for any 
k = 0, ...,£- 1 

j G (j(i) = kmodt. 

Thus we must have a(I^) C JW for any k = 0,...,£ — 1. Since a is a bijection, n I^'** = 
j(fc) n j(fc') = if k ^ k ' and \j < k <t-ii^ = Uo<a,<£-i J^-* = I n , the restriction of <r on I k is a bijection 
from to In particular caid(J^) = card(/( fc )) = r k . This means that we have the equivalences 

Vi fil ft >) (4fe)M ^ ^ < l(k) ) = J (k) f o r any k = 0, - 1. (27) 
\fc=0a=l * a / 
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Write Jw = (k,k + £, k + (r^ — !)£)■ Denote by o" € S n the permutation verifying 

a (i^) = k+(a-l)£ (28) 

for any k = 0,...,£ — 1 and any a = 1, r&. Let Sj(k) be the permutation group of the set 
The permutations a which verify the right-hand side of (|27p can be written a = o§t where r = 
(t(°\ r^" 1 )) belongs to the direct product Sj<p) X ••■ xS /( m) . We have then e(a) = e(a )(— l)^ 7 " ) x 

• • ■ x 

For any k G {0, ...,£ — 1} set 

T( fc ) eSj(fc) Va=l 4a / 

From (|22p and (|26p we derive 

£-1 ^ 
<M-PaO = e(«7 ) II x /( fe ) Pk - 

k=0 

Since ao(i^) = k + (a — 1)£, we can write by ([25]) <7QT^ k \i^) = k + (r( fc )(a) - 1)1 Thus we obtain 
p. - V r-lV( r )x" TCfe)(1) • • • x- T(fc)(rfe) - x~ P, ' k A ™ 

where ft, fe = (1,2, ...,r k ) and A /(fe) = n»<j ~ Xj/%i)- Finally, this gives 

l ~ l ( fe ) e ^ k 

fc=0 fc=0 



where for any k = 0, — 1, 



* + i + * * | i e - (1,2, ...,r k ) e (29) 



Theorem 3.2.1 Consider a partition [i of length n and £ a positive integer. For any k = 0, ...,£ — 1 
define the sets and as in [25]). 



If there exists k G {0, ...,£ — 1} such that card(/( fc )) / card(J^) then (pe(s^) = 0. 

Otherwise, for any k = 0,...,£ — 1, se£ = card(/( fc )) = card(J^) and define ao as in fggj) . 
Then each r^-tuple defined by A29\) is a partition and we have 



tPt( s ij) = £ ( a o)S^ tI = e(<T )char(y At ) 

where 1 ={/(°, I^}, ft) = (//°), and char(V^) is the character of the GL n - 
module = V(fx^) ® • • • ® V(Ai (<-1) ). 

Proof. One verifies easily from (|29p that each // fc ) is a partition. By the previous computation, 
we obtain 

tp e (A x x M ) = e(o- )Aj x 
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(with the notation of 12. 5[) . By definition of ip% we have also 

= e (°"o)H o ip e (A x x M ) = e(<r )H(Ai x = e^SVmw 



where the last equality follows from (JToJ). ■ 

Remark: The subgroup Gj appearing in Theorem 13. 2. H is characterized by X ={j(°, J^ -1 )}. This 
means that for type ^4, we have = 1^ for any > with the notation of 12.21 that is the sets 
contain only positive indices. 

Example 3.2.2 

Consider p = (1,2,3,4,4,4,6,6) and take £ = 3. We have \i + p 8 = (2,4,6,8,9,10,13,14). Thus 

= {3,5},/W = {2,6,7},/( 2 ) = {1,4,8} and = {3,6}, J« = {1,4,7}, j( 2 ) = {2,5,8}. T/ien 
//°) = (1, 1), = (1, 2, 2) and /i( 2 ) = (0, 1, 2). We have Gj ~ GL 2 x GL 3 x GL 3 . 

3.2.2 For G = Sp 2n 

We have p = p n = (1, 2, n). By using d2H) we deduce the identity: 

P„ = x[^ +1) ■ ■ ■ x ( n ^ +n) e(w)x~ w{l) ■ ■ ■ x„-"(") (30) 

where W is the group of signed permutations defined on J n = {n, 1, 1, n}, that is the subgroup 
of permutations w G Sj n verifying w(x) = w(x) for any x G J n . Given k G {0, ...,£ — 1} consider the 
ordering sequences 

/( fc ) = (j g J„ | p i + j = fcmodf) and J (fe) = (x G J n | x = kmodi). (31) 
Set p = | if I is even and p = ^ otherwise. 

The odd case £ = 2p — 1 Set ro = card(l^) and for any k = 1, — 1, Sfe = card(ifc), = 
card(i&) + card(Ig_fc). Write X^ k \ k = 1, for the increasing reordering of I& U Set I*- ** = 

(4 0) ,...,4° } ) and for fc > 

*(*> = (<<*>,. ..,«£). (32) 

This means that 1^ = (if , and = ...,4^). To simplify the computation, we are 

going to use the indices and the variables Xi,i G X^ rather than the variables Xi,i G J( & ) u J^~ k ) 
when G {1, — 1}. 
Consider 



^±1 | i g J(0)^ e and for ft > 0, = (sign(i) ^ + ^ | • G X«) g jru 



where for any i G J n , sign(z) = 1 if i > and —1 otherwise. For any z G /, we have = x^ w . 

Thus 

n *r w = n-r w n x r w ^^ +i) ---^=^u^ n ^ k 
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by definition of the /#)'s. Then (JSOJ) can be rewritten 

p,=4fii^- e-h n -r w n n *r° (o xff n ^ 



k=l w&V ig/(0) fc=lj eX (fc) fc=l igX (fc) 



This gives 



<PiW=4» n4 c S) >< e ( n *r° <0 n n *r w ~ fe 

fc=i tuew \i6/(°) fc=iiex( fc ) 

The contribution of a fixed w G in the above sum is nonzero if and only if 

i e ==> = Omod^ 

i G ==> w(i) = — kmodi for any k = 1, ...,p — 1 



(33) 



Thus we must have w(7 (0) U j( )) C and for any fc = 1, ...,p - 1, w(X( fc )) C J^" fc ). Recall that 
J (0) = j(°) and J [£ ~ k) = for fe = 1, ...,p - 1. Moreover 



p— l p— l 



J(°) Ul (0) |J I«UI W = J n and U |J J« U J^ fc ) = J n . 



k=l k=l 



Since the sets appearing in the left hand side of these two equalities are pairwise disjoint, we must have 
w(T (0) U/(°!) = J(°), and for k = 1, ...,p-l, w(X^) = J^~ k \ In particular card(j(°)) = 2card(/( )) = 
2ro and card(J^~ fc )) = card(X( fc )) = r& for any k = 1, ...,p — 1. We have the equivalences 



(34) 

Note that condition (ii) can be rewritten: w(X ) = JW for any k = 1, — 1. 
We can set = (— tq£, tqI) and for k = 1, ...,p — 1, 

= (_jfe _ afc 4 + ) g fc £) ) j{k) = (fc _ / g fc £ j fc + afc ^) 

with a fc + + 1 = r fc . 
Consider u>o € W defined by 

roo(ii 0) ) = offorae{l,..,r } (35) 
iA>o(«a ) = —A: — + (a — 1)£ for any k = 1, — 1. 

Denote by W the set of signed permutations w which verify (i) and (ii) in (f34|) . We have wq G W. 
Each w G W can be written to = u>ot> where v = (v(°> , r( p-1 )) belongs to the direct product 

W^(o) x x • • • x S X ( P ~i). Here Wj(o) is the group of signed permutations defined on 1^ U 1^ 

and for k = l,...,p — 1, S X (k) is the group of signed permutations defined on U X^ and 

verifying T^ k \X^) = X^K Indeed if T^ k \x) G X (fc) and x G X^, we would have w(x) G J^'^ and 
x G which contradicts (ii). This means that S X (k) is in fact isomorphic to the symmetric group 
S r . . Since the sets 1^ and X^ k \ k = 1, ...,p— 1 are increasing subsequences of J n , we have by Lemma 
\Zrne(w) = e(wo)(-lf v<0) \-l) 1 ^ x • • • x (-l)K" r(p_1) ). 
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Set 



p = e (-iy {vio)) vz n {i) I - d 

»(°)ew f(0) \ieJC°) 



P fc = 53 (-D' (r(fc)) 



\iGX( fc ) / 



We obtain 

(0) P_1 (&) 
k=l 

From (i23j) and (l35|) . we have u;oU^(ii° ) = t/°)(a)£ for any a = 1, ...,ro and 

w T {k) {i { a k) ) = —k — a k £ + (T^(a) - 1)£ for any a = 1, ...,r fe . 

This yields 

ro 

P = E (- 1 )'^ (0)) II >° )(a) = *~ifc*W and 
u(°)ew /(0) Q=1 



where for any fc = 1, ...,p - 1, ry r = — p r + (a^ + 1, ...,afc + 1) G Z rfc , 



a /( o)= n n o--*rx,) 

i<j ije/C) * r<s r,se/(°' 



and A X ( fe ) = | J (1 -) for any k = 1, — 1. 

i<j i,j£X( k ) 



Finally, this gives 



<^(^) = e(^o)^ (0 ) Pr ° Aj«» JJ a£ w J7rfe A x(fc ) = e(u) )^(o) > A /(0) JJ A x(fc) 

fc=i fc=i 

where 

M (0) =(^|ie/ (0) )-(l,...,ro)GZ^ (36) 

and for any k = 1, — 1 

^ fc ) = ^ S ign(i) MW + N | SlgnWA: | i € X< fc >) - (1, ...,r fc ) + («* + 1, ....,a* + 1) G Z'*. (37) 

Recall that the weights corresponding to the subgroup of Levi type Gj are written following the 
convention ([5]). 

Theorem 3.2.3 Consider a partition fi of length n and £ = 2p — 1 a positive integer. Let 1^ and 
jw be as in 157]). For any k = 1, — 1 define the sets and by 157]) and 
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• I/card(l(°)) 7^ icard(J^) or if there exists k € {1, ...,p - 1} such that card(X( fc )) / card(j( fc )) 
then tpi(s^) = 0. 

• Otherwise, set ro = card(I^) and for any k = 1, ...,p — 1, r^ = card(X^ c - ) ). Lei wq £ W be as in 
(Op. Consider (£) = pM, //fr" 1 )) waere i/ie /i^'s are denned fry (E6J) and ^/len 
(^) is a dominant weight of Pj withl ={I^°\X^'...,X^ P ~ 1 '} and we have 

Proof. The proof is essentially the same as in Theorem 13.2.11 We obtain 

<Pe(A x x 1 *-) = s(w )Ax x xu) 

where in the right-hand side of the preceding equality (^) is expressed on the basis {e^ | i € I n } (see 
(|10p ). This permits to write as in the case G = GL n 

<Pe( s ») = e(wo)H o ip e (A x a;^) = e(w )H(Ai x a?u)) = e(io )5(>) ):r 



Example 3.2.4 

Consider n = (1,2,3,4,4,4,6,6) and take £ = 3. We have //+p 8 = (2,4, 6, 8, 9, 10, 13, 14). Thus 1^ = 
{3,5},XW = {7,6,2,1,4,8} and = {6, 3, 3, 6}, = {8, 5, 2, 1, 4, 7}, = {7,4,1,2,5,8}. In 
particular ot\ = 2. Then = (1, 1) and /i^ = 

13-1 10-1 4-1 2 + 1 8 + 1 14 + 1 
— 1 + 3, 2 + 3, — -3 + 3, -L- -4 + 3, — -5 + 3, — 6 + 3 

= (-2,-2,-1,0,1,2) 

with the convention j2|). We have Gj ~ Sp± x GLq. 



The even case £ = 2p With the same notation as in the odd case, (|30p can be rewritten 

^=4f4£ p) n4S* e-h n *r w n *r (i)V n n ^ (i) - fe - 

fe=i wgiy ie/(°) iei(p) fc=ij e x( fe ) 

where 



This gives 



i 



^,)=^;<!n4w x e ^ ( n n *r (<H, n n *r (iM 

fc=i wew \ie/C°) i€/( p ) fc=ij e x( fc ) 

The contribution of a fixed w; € in the above sum is nonzero if conditions (|34f) are verified and 

i G /(p) => w (i) € J {p) . 

Since p = — pmod^ we have = = {—p — a p £, —p,p, ...,p + a p £}. This implies that w(I^ U 
1^) = and thus card(J^) = -card(jW). We then define wq by requiring ([33]) and u>o(«i P ^) = 
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p + (a — 1)£ for a £ {1, r p }. By using similar arguments as in the odd case, we obtain that w can be 
written w = wqv where v = (!)( ),tW, t^ p ~ 1 \ v^ p ) belongs to the direct product W^o) x S X (i) x • • 
• x S X (e-i) x Wj( p ) with Wj(p) the group of signed permutations defined on J(p) U l( p ). Note that W^p) 
is a Weyl group of type B Tp . By Lemma \2. 1.11 we have also e(w) = e(wo)(—l) l ( v °')(— l)'( r(1) ) x • • • x 
x We obtain 



fc=i «o»)eW( p) \iei&0 

The functions P&, = 0, ...,p — 1 can be computed as in the odd case. For P p , observe that each 
can be written = £cr according to the decomposition of Wj( P ) as the semidirect 
product (Z/2Z) rp oc S jW . We have then for any a = l,...,r p , wqv^X^) = £(a)(p + (<x(a) - This 
yields 



a=l 



rp 



Thus 

p » = n e (-i)"""" ("" , ■ n ) = 

where 

V), Br5 = n ( x - ^) n ( x - ^ n ^ - ^ 

i<j m'G/W r<s r,selM ie/to 

Indeed the half sum of positive roots is equal to (i, r p — ^) in type P rp . This means that when ^ is 
even 

p-i 

m p ») = e{w )x^ 0) A I{0) [[x x(k) A x(k) x a^ (p) A J(p)iBrp 

k=l 

where l,...,r p ). In particular the computation of ^(P«) makes appear positive roots 

corresponding to a root system of type B Tp . These roots do not belong to the root lattice associated 
with Sp2n- Hence, there cannot exist an analogue of Theorem 13.2.31 when I is even. With the previous 
notation, we only obtain: 

Proposition 3.2.5 Suppose G = SP2n and £ = 2p. 

• J/card(/(°)) + icard(j(°)), card(P») ^ icard(j( p )) or there exists k £ {l,...,p- 1} such that 
card(X( fc )) / card(jW) then = 0. 

• Otherwise, the coefficients appearing in the decomposition of cp^s^) on the basis of Weyl char- 
acters cannot be interpreted as branching coefficients and have signs alternatively positive and 
negative. 
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3.2.3 For G = SO 



2n 



As for G = Sp2n, the coefficients appearing in the decomposition of ^(s^) with t = 2p on the basis 
of Weyl characters cannot be interpreted as branching coefficients. Note that there is an additional 
difficulty in this case. Indeed, </?|(P^) cannot be factorized as a product of polynomials (1 — or) where 
(3 £ Z n . For example, we have for SO^ 



\ X\ 



1 - X\X2) = 1 + x 2 . 



This is due to the incompatibility between the signatures defined on the Weyl groups of types B and 
D when they are realized as subgroups of the permutation group Sj n . 

So we will suppose I = 2p — 1 in this paragraph. Recall that the elements of W are the signed 
permutations w defined on J n = {n, 1, 1, re} such that card({i £ I n \ w(i) < 0}) is even. Set 
Kn = {n — 1, 1, 0, 1, n — 1}. Each w £ W can be written w = (a according to the decomposition 
of W as the semidirect product (Z/2Z) n_1 oc S n . For any x £ J n , we have then £(x) = 1 if u>(x) > 
and £(cc) = —1 otherwise. Given w £ W we define w : J n — > -fT n such that w(x) = w(x) — £(x) for any 
x € J n . Then = u>(x). 

For type D n , we have p = p' n = (0, 1, re — 1) = p n — (1, 1). Hence 

w- p' n = wp n - (£(!),•», = (w(l),...,tfl(n)) = w- p n . 

Then we obtain 

P M = x{^ +0) • ■ ■ x^ +n ~ l) e(w)xf W ■ ■ ■ x~^ n \ 

For any k = 0, ^ — 1 set 

= (iGl n |^ + i- l = fcmod^) and J (k) = (x e K n \ x = kmod£). (38) 

We then proceed essentially as in 13.2.21 by using w instead of w and p' n = (0, 1, n — 1) instead of 
p„ = (1, ...,re). We only sketch below the main steps of the computation. 

Set ro = card(/(°)) and for any k = p — 1, Sfc = card(Ifc), = card(Ifc) + card(/^_fc). For 
k = 1, — 1, is defined as the increasing reordering of 1^ U I^~ k \ Consider 



I i £ /(°)J £ Z r '° and for fc > 0, 

(k) ( ■ + 1*1 - l + sign(i)A; A 
= I sign(z)-^ [ 1 £ X w J 



£ 



7T k 



We obtain 



MP,) =4% u4t >< e ( n n ^ (i) ~ fe 

fc=i wgvK \ieJC°) fc=ij S x( fc ) 

We also have the equivalences 

<p t [ n-r (i) n ft (o^^v^ 



g /(o) fc=ij g x( fe ) 



(ii) : {^(X^)) = j(* k "> for any k = 1, — 1 

(39) 
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We can write = (— (r — 0, (r — 1)£) and for k = 1, j>, 

j(/-fc) = (_£ _ ajfe 4 ^ + / g fr Q ) j(fc) = ( fc _ ^ ^ fc + afc ^) 

with afc + /3 fe + 1 = r k . Consider wo € W denned by 

wo(i ( a y ) = (a - IK f o r a G {1, M (40) 



= -k- a k i + (a - 1)^ for any fe = 1, - 1. 



Denote by W the set of signed permutations w GW which verify (i) and (ii) in (|39p . We have wq 6 W. 
Each ro G >V can be written w = wqv where v = (v^-°\ r^ 3 " 1 ') belongs to the direct product 

Wj( ) x S X (i) x • • • x S X ( p -i) with Wj( ) the Weyl group of type D ro defined on 1^°' U We have 
by Lemma EXUeH = e(wo)(-l)'^ (()) )(-l) /{T<1)) x ■ ■ • x (-l)^" 1 )). 
Set 

Po = e ( n ^ (0)w 

^ = E (-!)' (T(fc)) ^ ( II x^W"* J for any k G {1, ...,p - 1}. 

r«GS x(fc) \iexW / 

We obtain 

(0) P_1 (ft) 

<Pt( p n) = e(wo)a^ (0) P Yl x x(k) P k . 

k=l 

Given G W}(o), we define w^ ^ = — £ v where £ v (i a ) = 1 if v{i a ) > and —1 otherwise. By P0|) . 
we have for any a = 1, ...,ro, wo^H^) = "^°H a K- Moreover since € £x( fc ) 

w T^(i^) = -k - a k £+(T^(a) - l)£ for any o = 1, ...,r fc . 

This yields 



p = e (-i)^ (o)) n^r (a) =^ o A /( o)and 



e (-i)' (r(fc)) n^r (a)+K+i) =4 r ( fc) A; 



where for any k = 1, ...,p - 1, r? = -p' + (a fc , ...,£**.) = -p r , + (a* + 1, ...,Q! fc + 1) G Z r ' fe , 



A/ ( o) = J| (1-^) II (l-x r x s )and 



A XC=) = II (1 - for any A; = 1. 

i<j,i,jex( k ) 



This gives 



= £ ( w o)V) Pr ° A J(0) JJ a£ (Jk) ??rfc A X ( fc) = e(u;o)x^ ( ( ) ) A /(0) JJ zJa^j 
fc=l fe=i 
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where 

^ (0) = {^T 1 I i 6 ^ (0) ) - (0,-,ro - 1) 6 Z", (41) 
and for any k = 1, — 1, 

=1 i € X<*> fsign(i) ^ N | z G - (0, r fe - 1) + (a fe , a k ) G Z'\ (42) 

Note that these formulas are essentially the same as for G = Sp2 n , except that we use p' n = (0, n — 1) 
instead of p n = (1, ...,n) for the half sum of positive roots. This gives the following theorem whose 
proof is identical to that of Theorem 13.2.31 

Theorem 3.2.6 Consider a partition fi of length n and £ = 2p — 1 a positive integer. Let 1^ and 
be as in For any k = 0, ...,p - 1 define the sets X<» and jW by {22P and [Ml . 

• // card(jW) / |(card(j(°)) + 1) or if there exists k G {1, ...,£> - 1} such that card(X( fc )) ^ 
car d(jW) then tp e ( Sfl ) = 0. 

Otherwise, set ro = card(I^) and for any k = 0, — 1, r^ = card(X( fc )). Lei u>o € W 6e as 
Consider (£) = (//("), jU^ 1 )) w/iere tfie 7 s are denned oy gip and g§). T/ien (£) 
is a dominant weight of Pj with! ={l(°\X( 1 \.., ^} and toe /iaue 



in 



Example 3.2.7 

Consider // = (1, 2, 3, 4, 4, 4, 6, 6) and ta/ce £ = 3. We have fi + p' 8 = (1, 3, 5, 7, 8, 9, 12, 13). Thus = 
{2,6,7}, A^ 1 ) = {8,4,1,3,5} and = {6, 3, 0, 3, 6}, jW = {5,2,1,4,7} and j( 2 ) = {7,4,1,2,5}. In 
particular a\ = 2. Then p,^ ' = (1,2,2) and 

(i) ( 13-1 1 , _ 7-1 1-1 „^„5 + l 8 + 1 
F J = I 3 1 + 3, 2 + 3, 3 + 3, — 4 + 3, — 5 + 3 

= (-2,-1,0,1,1) 

We have G x ~ S0 6 x GL 5 . 
3.2.4 For G = S0 2n +i 



Set L n = {n — 1, 1,0, 1, n}. Each u; G W can be written u; = £er according to the decomposition 
of W as the semidirect product (Z/2Z) n oc # n . Given w G W we define n; : J n — » L n such that 
u>(a;) = w(x) + ^(1 — £(x)) for any x G J n . For any y G L n , set y* = y + 1. We have then u5(af) = 
(tu(ar))* = + 1. 

Observe that p = p'^ = (|, |, n — |) = p n — (|, |). Thus 

w ' Pn = w ■ p n - -(£(l),...,£(n) = (n5(l),...,n7(n)) - -(1,...,1). 
This permits to write 

P a = . . . 4f" +n) J2 e(w)xf {l) ■ ■ • s^H (43) 

For any k = 1, £ (observe that does not run over {0, — 1} as for G = Sp2 n or SC^n) set 

/( fe ) = (i g I n | fi { + i = kmodl) and J (fc) = (x G L n | x = kmodt). (44) 

Note that (J^)* = We then proceed essentially as in l3.2.2l bv using w instead of w. We are 

going to see that for G = SC^n+ij there exists an analogue of Theorem 13.2.31 whatever the parity of I. 
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The even case £ = 2p For any k = 1, set s k = caid(I^), r k = card(/( fc )) + card(/^ fc+1 )) and 
define as the increasing reordering of I {k) U Set 

*« = (#>,..., 4?)- (45) 
For /c = 1, ...,p consider the r^-tuple such that 

(k) ( ■ + N +sign(i)fc - 1+sl | n W \ 
/x w = I sign(z)^ | % G X w j G Z rfe . 

For any i € with fe = 1, ...,p, we have x i w ^ 1 = x~ w<yl \ Thus 

n ^ = n *r* (0 n ^ = n -r^- 1 n ^ 

and by definition of the H&)% (USD can be rewritten 

^=n«x E-wn n *r (i) *n n *r* +i - 

fc=l wGW fe=lj g x( fe ) fc=lj G Y(fe) 

We obtain 

=fi4i >< e ^ (n n *f ( ^ k+1 

k=i w&w \k=iiex( k ) 

We deduce the equivalences 

V s k \ 

ft II xT^+X- 1 ^ = J^ fc+1 ) for any k = 1, ...,p. (46) 

In particular we must have car 

= card(jW) = r fc . We can write 
j(/-fc+l) = + i _ akif -A: + 1 + and J (fc) = (k - k £, k + a k £) 
with afc + + 1 = rfc. Consider wo € W defined by 

w Q (iW) = -k + 1 - a k £ + (a - 1)£ for any k = 1, (47) 



Denote by W the set of signed permutations w; G which verify the right-hand side of (146D , We have 
^0 G W. Each w G W can be written it; = tfo^ where r = (tW, ...,r( p )) belongs to the direct product 
S xW x • • • x S X(P) . We have also by Lemma &JJ]e{w) = e{w )(-l) l( - T(1 ^ x • • • x (-l)'( r(p) ). 
For any k = 1, set 



p fe = e (-i/ (r(fc v n 



^-{o T( fc )(i)-fc+l 



We obtain 

= £ ( w o) n x xw Pk - 

k=l 
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By (|47p we have 

w rW(iW) = -k + l-a k £ + (t^(o) - 1)4 for any a = 1, ...,r fc . 

This yields 



^ = e (-D' (r(fc)) ft ^ (fe )(a)+(Qfe+i) = a x( , , 



r^es x(k) 0=1 



where for any /c = 1, ...,£>, r/ r = — p r + (a*. + 1, at + 1) € Z rfe and 



X 
X 



i<j i,j£X( k ) 



Note that the computation only makes root systems of type j4 appear in this case. This gives 

p (fc) _ p 
^(-Pm) = £ ( u; o) n x xw rkA xw = e(wo) II x^ w A x(k) 

k=l k=l 

where for any k = 1, •••,£>, 

M (fe) = Isign(i)^^ t^J. | iGX (fc)j _(i ) ..., rfc ) + (a fc+1 ,....,a Jfe+1 ) eZ'*. (48) 

Similarly to Theorem 13.2.31 we obtain: 

Theorem 3.2.8 Consider a partition of length n and t = 2p a positive integer. For any k = 1, ...,p 
define the sets X<- k \jW by and 

• If there exists k € {1, such that card(X( fc )) ^ card(j( fc )) then ip^Sa) = 0. 

• Otherwise, for any k = l,...,p, set = card(X( fc )). Let wq £ W be as in ( f^7| ). Consider 
Of) = (flP~\ ...,/x^) where the 's are defined by |^<g| ). Then Of) is a dominant weight of Pj 
withl ={X (1 \..,X^} and we have 

<Pt( s n) = £ (wo)S^j. 

Example 3.2.9 Consider fi = (2,5,5,6,7,9) and £ = 2. Then fi + p 6 = (3,7,8,10,12,15). Hence 
h = {1,2,6} and I 2 = {3,4,5}. Moreover J 2 = {4,2,0,2,4,6} and J x = {5,3,1,1,3,5}. Then w 
sends X\ = {6,2,1,3,4,5} on J 2 - This gives 

~_/65432ll23456 
W0_ V453T20 1 3 2 4 6 5 

by using Hence 







5 4 3 


2 1 


12 3 4 


5 6 \ 




w ° u 


6 4 2 


3 1 


13 2 4 


6 5 J ' 


We have e{fx) = 1, 


a\ = 2 and 










^ = (" 


-7,-3,-1,4,5,6) - 


-(1,2,3, 


4,5,6) 


+ (3,3,3,3,3, 


3) = (-5 


We have then Gj : 


- GLq. 
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The case £ = 2p + l In addition to the sets X^ k \ k = 1, ...,p defined in (|45p . we have also to consider 
jiP+l) = This yields to define 



M (P+1) = ( ti ±j + P | { e j(p+iA 



We have 



x. 



fe=i uieVK \ie/(p+ 1 ) fc=iiex( fe ) 

and the equivalence 



-w(i)- P yj 77 -«(i)+fc-i | _i_ n / = ^ for any k = 1, ...,p 

\ ©(jOh-D u T (p+1) ) = J(^rf = J&h-D 



i n ^ p n n **~ wW+ + ° 

tg/(p+i) fc=l; e x(*0 / 

(49) 

Indeed (j( p+1 ))* = In particular we must have card(j( p+1 )) = 2card(/( p+1 )) = 2r p+ i. Thus we 

can set = (— p — (r p +\ — 1)£, —p + r p+ i£). Consider wq G W defined by and 

^o(^i P+1 ^) = —P + for any a = 1, ...,r p+ i. (50) 



Denote by W the set of signed permutations w £ W which verify the right-hand side of (|49p . We have 
wq S W. Each u> € W can be written w = wqv where v = (t' belongs to the direct 

product x • •• x S X ( P ) x Vt^-fp+i). We have also e(w) = e(wo)(-l) l ^ T<1> ^ x • • • x (— 1) / ( t(p) )(-1)'( 1 ' (p+1) ). 
This permits to write 

( +i) P (fe) 
<M^) = ^(^o)^J +1) Pp+i Yl x xik) p k where 

fc=i 

p p+1 = £ ( n .r o,;(p+1)(i) - p 

The functions -Pfc,fc = 1, ...,p can be computed as in the even case. For P p +i, observe that each 
V (p+V ^ Wj( p+ i) can be written i>(p +1 ) = (a with a E S^p+i). According to this decomposition we 

have for any a = 1, ...,r p+ i, wqv^ 1 ^^^) = £(a)(— p + a(a)£). 



r>+i \ 

e(o)(-p+<r(o)<)-p 



a=l 



r " +1 i-e(«) 



e (-o^-^n- 



2 £°"( a ) 



Thus 



^= n -r i/2 e (-i)^ Cp+1)) k +1) -n^ =^t iA ^) 



ie/(p+!) u(p+ 1 )ew/ /(p+1) V o=l 
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■n) 

i<j iJeliP^ 1 ) r<s r,sG/(P+ 1 ) *€/Cp+ 1 ) 



i<j i, 

This means that when £ is odd 



fe=i 



where 

/' 



This gives the following theorem: 

Theorem 3.2.10 Consider a partition \x of length n and I = 2p+l a positive integer. Define X^ k \ 
k = l, ...,p and J(P+!) by g^j and g3p. 

• 7/card(/( p+1 )) / ^card( j( p+1) ) or if there exists k € {1, ...,p} such that cardpf( fc )) / card(J^) 
then tpg(Sfj) = 0. 

• Otherwise, set r p+ i = card(7( p+1 )) and for any k = l,...,p, r^ = card(X( fc )). Let wq G W 
verifying fl^Tp and < f50j) . Consider (j) = (jti^'jfiW,...,^) where the fj,^ 's are defined by 
and (E2P- T/ien ( p ) is a dominant weight of P+ with 1 ={I^ p+l \X^\ ...,X^} and we 
have 

Example 3.2.11 Consider u = (1, 5, 5, 6, 7, 9) and take £ = 3. We have \i + p 6 = (2, 7, 8, 10, 12, 15). 
77ms JfW = {4,2,5,6},/( 2 ) = {1,3} and jW = {5, 2, 1,4}, j( 2 ) = {4,T,2,5}. In particular a 2 = 
1. T/ien 

m / 10-1 7-1 12 15 \ , 

= ( 3 1 + 2, — -2 + 2,--3 + 2,y-4 + 2j =(-2,-2,3,3) 

and // 2 ) = (^t^ — 1, — 2) = (0, 1). Moreover, one has by using (11) 



3 '3 
Hence 



654321123456 
305421235614 



654321123456 
"'" l 4l6532235614 

and e(/i) = 1. We have moreover G% ~ SO5 x GL4. 



4 Parabolic Kazhdan-Lusztig polynomials 

We recall briefly in this section some basics on Affine Hecke algebras and parabolic Kazhdan-Lusztig 
polynomials associated with classical root systems. The reader is referred to [14J and [16J for detailed 
expositions. Note that the definition of the Hecke algebra used in p3] coincides with that used in [9] 
and [16] (with generators H w ) up to the change q —>■ q^ 1 . 
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4.1 Extended affine Weyl group 

Consider a root system of type An-i, B n ,C n or D n . For any f3 G P n Z ra , we denote by ig the 
translation defined in fjjjj by 7 1 — ► 7 + /?. The extended affine Weyl group W is the group 

W = {wtp \weW,(3eP} 

with multiplication determined by the relations tpt^ = i^ +7 and wtp = t w .pw. The group W is not 
a Coxeter group but contains the affine Weyl group W generated by reflections through the affine 
hyperplanes H a ^ = {/3 G fjj£ | (/?, a v ) = &}< It makes sense to define a length function on W verifying 

Kwtp) = \(P,a v ) + l R _(wa)\ (52) 

where for any w G W, 1r_ (w ■ a) = if w ■ a G P+ and 1r_ (w ■ a) = 1 if w ■ a G — P+ = P_. Write 
ng for the element of maximal length in WtpW . It follows from (|52p that for any A € P+ we have 
l(wt\) = l(w) + l(t\). This gives 

™A = w t\ (53) 

where wo denotes the longest element of W. There exists a unique element 77 G P+ such that the 
fundamental alcove 

A={0€t)n\ (/3, a V ) > Va G P+ and (/?, r? v ) < 1} 

is a fundamental region for the action of W on f)Jj. This means that, for any G f)^, the orbit • /3 
intersects *4 in a unique point. Each w (zW can be written on the form to = w^Wgs where w a s G 
and belongs to the stabilizer of .4, under the action of W. This implies that A is also a fundamental 
domain for the action of W on f)J. The Bruhat ordering on W is defined by taking the transitive closure 
of the relations 

w < sw whenever l(w) < l(sw) 
for all w G W and all (affine) reflections s G W. 

In fact the natural action of W on the weight lattice P obtained by considering P as a sublattice 
of fjjg is not that which is relevant for our purpose. For any integer m G Z* we obtain a faithful 
representation 7r m of on P by setting for any j3, 7 G P, w G 

vr m (w) • 7 = to ■ 7 and n m (tp) -7 = 7 + m/3. 

Warning: In the sequel, the extended affine Weyl group acts on the weight lattice P via 7r_^ 
where £ is a fixed nonnegative integer. 

We write for simplicity wtp ■ 7 rather that TV^i{wtg) ■ 7. Hence for any w G W and any (3 G P, we 
have ■ 7 = w ■ 7 — £w ■ @. The fundamental region for this new action of W on P is the alcove Ae 
obtained by expanding A with the factor —I. This gives 



M = 



( {" = 






> 


V\ 


> ■ ■ 


> 


v n > 


—£} for types A 


■ { r 


iyu ■ 




> 


V\ 


> ■ ■ 


> 


Vn > 


-1/2} for types B,C 


{u = 


iyi, . 




> 


V\ 


> V2 


> 


■ ■ > 


v n > -e/2}u 


( {» = 


(vi, ■ 




> 


V\ 


> V2 


> 


■ ■ > 


v n > for type D 



Consider a weight /3 G P. Then its orbit intersects An in a unique weight v. Then there is a unique 
w(f3) G W of minimal length such that w(f3) ■ v = j3. We denote by W v the stabilizer of v G At in W. 
Since v G Ae, W u is in fact a subgroup of W. 
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Lemma 4.1.1 Consider A G P + and suppose i > n. Then 

1. w{£X + p) = n\*T~ n+1 with A* = — wq{\) and r = s\S2 ■ ■ ■ s T -\t £1 for type A. 

2. w{£X + p) = n\ for types B, C and D. 

Proof. 1 : See Lemma 2.3 in [9]. 

2 : Observe first that u>q • p = —p belongs to At for types B, C, D since £ > n. We have 
IX + p = i_ A • P = t-\w ■ (u>o • p) = t-\w ■ (-p). 

Moreover W~ p = {1}. Since —p G At this means that w(£X + p) = t^\wo = Wot WQ .r\\ = w ot\ = n x 
where the last equality follows from ([53]) . ■ 

4.2 AfRne Hecke algebra and K-L polynomials 

The Hecke algebra associated with the root system R of type A n , B n , C n or D n is the Z[q, (7 _1 ]-algebra 
defined by the generators T w ,w G W and relations 

T Wl T W2 = T Wl T W2 if l(w\Wv) = l(wi) + l(w 2 ), 
T S T W = (q^ 1 - q)T w + T SiW if l(siiv) < l(w) and i G I n . 

In particular we have Tf = (q -1 — g)T, + 1 for any i G I n . The bar involution on H is the Zdinear 
automorphism defined by 

q = q~ l and T w = T~ x for any w G W. 
Kazhdan and Lusztig have proved that there exists a unique basis {C' w \ w G W} of H such that 

Cu> = C'w an< i = 5^ Py,wT y 
y<w 

where = 1 and p y>w G q1>\q\ for any y < w. We will refer to the polynomials Py jW (q) as Kazhdan- 
Lusztig polynomials. They are renormalizations of the polynomials P y>w originally introduced by 
Kazhdan and Lusztig in [6]. Namely W6 have Py^w — q^ w ^ Ks/) Py^. 
Let us define the g-partition function V q by 

Given A and p in P, the Lusztig (/-analogue K\„(q) is defined by 
Then one has the following theorem due to Lusztig: 

Theorem 4.2.1 Suppose A, p are dominant weights. Then Kx ^q) = Pn M ,n M (?)- 
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One defines the action of the bar involution on the parabolic module P v = Hi;, v G Ag, by setting 
q = q^ 1 and h ■ v = h-u for any h G H. Deodhar has proved that there exist two bases {C^ | A G W-u} 
and {C^ | A G W ■ u} of P v belonging respectively to 

L+ = Z[q]X and L~ = Zfe" 1 ]* 

characterized by 

A ^a and i 

= AmodgL+ \ = \modq~ 1 L~ 

We will only need the basis {C^ | A G W-v} in the sequel. The parabolic Kazhdan-Lusztig polynomials 
P^T are then defined by the expansion 

Cx = E (-ly^+^p^q- 1 ^ 

tieW-x 

(see [16J Theorem 3.5). In particular they belong to Z[q]. Their expansion in terms of the ordinary 
Kazhdan-Lusztig polynomials is given by the following theorem due to Deodhar: 

Theorem 4.2.2 Consider v G Ai and A G W ■ v. Then for any /x G W ■ A we have 

P x,M = E (-9)' (z W)^(A)(9) (54) 

with the notation of \^.l\ 

Remark: When v is regular, that is W u = {1}, we have ' (q) = P w (h),w(X)(q)- 

5 Generalized Hall-Littlewood functions 
5.1 Plethysm and parabolic K-L polynomials 

Consider t a nonnegative integer and £ G C such that £ 2 is a primitive ^-th root of 1. We briefly recall 
in this paragraph the arguments of [9] which establish that the coefficients of the plethysm ip^{s\) on 
the basis of Weyl characters are, up to a sign, parabolic Kazhdan-Lusztig polynomials specialized at 
9=1. 

For any A G P+, denote by V q (X) the finite dimensional f7 9 (g)-module of highest weight A. Its character 
is also the Weyl character s\. Let U q> z(g) be the Z[q, <7 -1 ]-subalgebra of U g (g) generated by the 
elements 

where Ei, Fi, K^ 1 , i G I n are the generators of U q (g). The indeterminate q can be specialized at ( in 
Uq t z(o). Thus it makes sense to define U^(g) = U q> %{g) <8>z[ ?) g-ii C where Z[q, q^ 1 ] acts on C by q i— ► (. 
Fix a highest weight vector v\ in V q {X). We have V q (X) = U q (g) ■ v\. Similarly V^(A) = U^(g) ■ v\ is a 
f/f (jj) module called a Weyl module and one has char (V£ (A)) = s\. The module V^(A) is not simple 
but admits a unique simple quotient denoted by L(X). 

From results due to Kazhdan-Lusztig and Kashiwara-Tanisaki one obtains the following decomposition 
of char(L(A)) on the basis of Weyl characters: 
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Theorem 5.1.1 Consider A G P + . 

1. For I sufficiently large, the character of L(X) decomposes on the form 

char(L(A)) = i)'(-(a+p))-K-(m+p)) ^ a+ ^ (1) ^ (55) 

where the sum runs over the dominant weights p, G P+ such that p, + p € W ■ (A + p). 

2. The parabolic Kazhdan-Lusztig polynomials P p+p \+ p (q) have nonnegative integer coefficients. 
Remarks: 

(i) : The decomposition (|55p has been conjectured by Kazhdan-Lusztig and proved by Kashiwara- 
Tanisaki. In [5], Kashiwara and Tanisaki have also obtained that the parabolic Kazhdan-Lusztig 
polynomials have nonnegative integer coefficients as soon as the Coxeter system under consideration 
corresponds to the Weyl group of a Kac-Moody Lie algebra, thus in the particular context of this 
paper. 

(ii) : In Assertion 1 of the above theorem, the integer i is only explictely known in the simply laced 
case (see Theorem 7.1 in |17|). 

Consider a nonnegative integer t. The Frobenius map Fr^ is the algebra homomorphism defined 
from U c (q) to U(g) by Fr^(ilQ) = 1 and 

Fr t (rfQ) = { ^ if 1 dMdes k and Fr,(F (fc) ) = { ^ if 1 divides k 
' 10 otherwise ' (0 otherwise 

where &i, fi, i E I n are the Chevalley generators of the enveloping algebra U (q). This permits to endow 
each [/(g)-module M with the structure of a [/^(g)-module M Fre . Then we have 

char(M Fr ^) = ^(char(M)) 

in particular for any A € P+, ch&v(V(\) Fre ) = ip e (s\). 

r q r q 

Each dominant weight A G -P+ , can be uniquely decomposed on the form A = A + i\ where A, A G -P+ 

r r r r r 

and A = (Ai, A n ) verifies < Aj + i — X L < I for any i G I n . 

Theorem 5.1.2 (Lusztig) The simple U^(g)-module L(\) is isomorphic to the tensor product 

L(A) ~ L{\) ® T/(A) Fr *. 

r q 

By replacing A by £X in the previous theorem, we have A = and A = A. Thus L(£X) ~ V(X) Fli . Then 
one deduces from (|55l) the equality 

Msx) = char(L(A)) = £ (-1)'(-(a+p))-'(-(^p)) P - + ^ (1) ^ 

which shows that the coefficients of the expansion of ipi{s\) on the basis of Weyl characters are, up 
to a sign, parabolic Kazhdan-Lusztig polynomials specialized at q = 1. This gives 

l< ^e(s\),Sn >\ = \<s x , <p(sp) >\ = P~ +p/x+p (l)- 
By definition of the action of W on P we have W ■ (IX + p) = W ■ p. This implies the 
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Corollary 5.1.3 (of Theorems \4-2.2\ and \5.1.!ty . For £ as in 1 of Theorem\4-2.2\ we have 



Msx) = £ (-l)^ A+ ^-^+'» P; +p/x+p (l)s». 

In particular (p(s p ) 7^ if and only if p + p G W ■ p, that isp + p = w- p — 1(3 with w and (3 G P. 

Remark: The equivalence ^(s^) 7^ <^=^ p + p G W ■ p can also be obtained more elementary from 
algorithms described in[ 



5.2 Parabolic K-L polynomials and branching coefficients 

Warning: In the sequel of the paper we will suppose i sufficiently large so that assertion 1 of Theorem 
\4-2.2 holds. Moreover I is assumed odd when the Lie groups under consideration are of type C or D. 



Under these hypotheses we have for any p G V n ^>e(s p ) = or 

V^(sm) = e (^o)5 , (M) jX (56) 

according to the results of 13.21 

Remark: According to the algorithms described in 13.21 when <^(,s M ) ^ 0, the cardinalities of the sets 
j( k ) or contained in I are determined by those of the sets j( k \ In particular they depend only 
on n and I and not on the partition p considered. Thus in (|56p . the underlying subgroup of Levi type 
Gj is, up to isomorphism, independent on p. 

By using Proposition 12.5.21 and Theorems 13.2. H [3.2.31 13.2761 13.2.81 13.2.101 we deduce from Corollary 
15X31 the 

Theorem 5.2.1 For any A, p G V n such that p + p G W ■ p 

where (^) and I are obtained from p and i by applying the algorithms described in \3.2\ 
5.3 The functions Hf, 

For any ft G P n , we define the function by setting 

\ev n v ' 

where for any A G V n , [V(X) : Vj(^)] q = P p+p t\+ p {q)- We also consider the function such that 

T e - , 



K = GL- (58) 



Theorem 5.3.1 Consider a partition p G V n . 

1. The coefficients of G p and H on the basis of Weyl characters are polynomials in q with non- 
negative integer coefficients. 
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2. We have = 



3. For £ sufficiently large H„ = Q' p , that is H„ coincide with the Hall-Littlewood function associated 
with p. 

To prove our theorem we need the following Lemma: 
Lemma 5.3.2 Consider (3 £ Z n . 

• In type A n ~i, suppose £ > n. Then the weight £(3 -\- p is regular. 

• In type B n , C n or D n , suppose £ > 2n. Then the weight £(3 + p is regular. 

Proof. Consider w € W and t 7 such that t 7 w-(£p+p) = £(3+ p. Then 5 = £(3+p-w-(£(3+p) G {£Zf '. 
Set (3 = (Pi, . ..,{3 n ). For any i = 1, ...,n, the i-th coordinate of 5 is 5i = £f3 i + p i — £(3 W ^ — p w ^y 



Since 5i G £Z, we must have 



Pi - Pw{i) 



G £7L. One verifies easily that for type A n -i, 



Pi - Pw(i) 



< 



n — 1 and for types B n ,C n , D n 



Pi ~ Pw{i) 



< 2n. Hence when the conditions of the lemma are 



verified. 



Pw{i) 



for any i = 1, ...,n. This gives w = 1 . The equality t^w ■ (£(3 + p) = £(3 + p 



implies then that 7 = 0. Thus the stabilizer of £(3 + p is reduced to {1}, that is £(3 + p is regular. 
Proof, (of Theorem 15.3. lj) 



1 : Follows from Theorem 15.1.11 and (157p . 

2 : When £ = 1, we have seen that G = Gj and (f) = p. Thus [V(X) : V x (f)} q + only if A = p. In 



this case H}. 

H 1 



for [V(X) : Vj{$] q = [V(X) : V(X)}, 



1. 



3 : Suppose £ as in the previous lemma. We have [V(X) : Vj( = P^+n ex+pil)- Since £X+p is regular 
for the action of W, we obtain by Theorem l4.2.2l Pf„ +p £ X+p (q) = Vw(ip+p),w(i\+p) (l)- By using Lemma 
14-1-lj we deduce ^ +Pj£A+p (?) = Pn„,n x {q)- Now b y Theorem [OH this gives P^ +piX+p {q) = K x ^(q). 
Finally 

K = E [^( A ) : = E K ^(9>x = Q'p- (59) 



xev n 



Remarks: 

(i) : By the previous theorem the functions H p interpolate between the Weyl characters and the 
Hall-Littlewood functions. 

(ii) : When £ is even for types C and D, one can also define the functions G 1 ^ and H p by setting 
G l p = J2 Xe v n i V+p/A+ P (^) s A and H p = G £t,i respectively. When £ > 2n we have yet H* = Q' p , but 
the polynomials P~ +p i X+p (q) cannot be interpreted as quantizations of branching coefficients. 

(iii) : The conditions £ > n for type A n _\ and £ > In for types B n ,C n , D n appear also naturally in 
the algorithms of 13.21 When they are fulfilled, one has ipi(st p ) = 0, or = Ik for any k = 1, n and 
J k = I k = for k <£ {1, ...,n}. Then [{£p)/£) = p and Gj = H. Hence [V(X) : Vx{fj\ = K Kp for any 
A G V n . This yields equality ([59]) specialized at q = 1. 
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6 Further remarks 



6.1 Quantization of tensor product coefficients 

Consider fi G V n and set [i = (^°\ fi^-i)} as in Theorem 13.2.11 For G = GL n , the duality 
( _ = [V(X) : Vj(fy] yields a g-analogue of the Littlewood-Richardson coefficient . ( _ } 

defined by setting 



"( At (o),..., /i ^-i') wy L r v v \£ J fj.+p,ex+p 

By Theorem 15. 1.1[ , ■. (q) have then nonnegative integer coefficients. 

(n* > ,...,p _i ) 

In [10] ; we have shown that there also exists a duality between tensor product coefficients for types 
B,C,D defined as the analogues of the Littlewood-Richardson coefficients by counting the multiplic- 
ities of the isomorphic irreducible components in a tensor product of irreducible representations and 
branching coefficients. These branching coefficients correspond to the restriction of S02 n to subgroups 
of the form SO2r x • • -S02 Tp where the rj's are positive integers summing n. These subgroups are 
not subgroups of Levi type, thus the Littlewood-Richardson coefficients for types B, C, D cannot be 
quantified as in ([60f> by using parabolic Kazhdan-Lusztig polynomials. 

For G = S02n+i, Sp2n or S02n and A G V n , denote by 23(A) the restriction of the irreducible finite 
dimensional GL^r-module of highest weight A to G. Consider a p-tuple {^°\ of partitions 

such that G V Tk for any k = 0, ...,p — 1. One can define the coefficients £>^ (0) ^(p-i) as the 

multiplicity of V(X) in 23(//°)) ® ■ ■ ■ ® ^(/i^ 1 )), that is such that 

2J(/i (0) ) ® • • • ® 33(m (p_1) ) ^ K(A) eS MW,..,^-i) . 

We have also obtained in [TO] a duality result between the coefficients £>^ (0) ( p _i) and branching 
coefficients corresponding to the restriction of G to the subgroup of Levi type GL ro x • • • x GL Tp l . 
The coefficients 2? ( ) can be expressed by using a partition function similarly to Proposition 

12.5.11 By quantifying this partition function, one shows that they admit nonnegative ^-analogues. It is 
conjectured that stable one-dimensional sums defined in [4] from affine crystals obtained by considering 
the affinizations of the classical root systems are special cases of the ^-analogues obtained in this way. 
Recall that the subgroups of Levi type Gj obtained in the theorems of 13.21 are, up to isomorphism, 
determined only by G and I. This implies that there exist subgroups of Levi type L in G which are 
not isomorphic to a subgroup Gj. This is for instance the case when G = Sp2 n for the subgroups of 
Levi type Gj ~ GL ro x • • • x GL rp _ 1 such that > 1 for any k = 0, ...,p — 1. Indeed, by Theorem 
13.2.31 when tq = card(/(°)) > 1, G% is isomorphic to 

Sp2ro x GL ri x • • • x GL Tpl . 

This implies that one cannot obtain in general a quantization of the tensor product coefficients 
S) (o) (p-i) by using parabolic Kazhdan-Lusztig polynomials as in ([60}) . 

6.2 Combinatorial description of the functions 

When G = GL n , the functions defined in (j57[) admit the following combinatorial description 

TeTab e (fj,) 
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where Tab^(^) is the set of £-ribbon tableaux of shape fi on /„ and s the spin statistic defined on ribbon 
tableaux (see [8] page 1057). Recently, Haglund, Haiman and Loehr have obtained the expansion of 
the Macdonald polynomials in terms of simple renormalizations of the LLT polynomials G . This 
expansion yields a combinatorial formula for the Macdonald polynomials [3]. 
This suggests to investigate the following combinatorial problem: 

Problem 6.2.1 Find a combinatorial description of the polynomials G^ and the q-analogues [V(X) : 
Vj{j)]q related to the roots systems of type B,C or D. 

6.3 Exceptional root systems 

It is also possible to define the plethysm ip e and the dual plethysm ip e for exceptional root systems. 
Consider such an exceptional root system R and fi a dominant weight for R. Denote also by s^ the 
Weyl character of the irreducible finite dimensional module of highest weight A. When £ is sufficiently 
large (the bound depends on R), we have ^(s^) = s^. For the other values of £, one shows that the 
polynomial <^i{e^Y\ aeR+ {l — e a )) do not factorize in general as a product of factors (1 — x@) where 
/? is a positive root. This implies that one cannot define generalized Hall-Littlewood functions for 
exceptional types by proceeding as in ([58]) . 

6.4 Stabilized plethysms 

When G = Sp2 n or S02 n and £ is even, we have seen that the combinatorial methods of Section [3] do 
not permit to obtain the coefficients of the expansion of the plethysms tp(s\) on the basis of the Weyl 
characters. In |12j . we show that this difficulty can be overcome by considering stabilized power sum 
plethysms, i.e. by assuming n > £ |A| . Under this hypothesis, one can indeed prove that the coefficients 
in the expansion of (p(s\) coincide for G = S02n+i, Sp2 n and S02 n - So it suffices to compute them in 
type B n for which we have a relevant combinatorial procedure in both cases £ even and £ odd. 

Note: While revising a previous version of this work [11], I was informed that Grojnowski and Haiman 
J1J/ also define, in a paper in preparation, generalized Hall-Littlewood polynomials for reductive Lie 
groups. Their polynomials are introduced as formal q-characters depending on a subgroup of Levi 
type. The coefficients of the corresponding expansion on the basis of the Weyl characters are also 
affine parabolic Kazhdan-Lusztig polynomials. As far as the author can see, the generalization of the 
Hall-Littlewood polynomials presented in the present paper satisfies the general definition given in fEj 
(see Definition 5.12). Nevertheless, our combinatorial results based on the study of the power sum 
plethysms on Weyl characters are completely independent of the approach of Grojnowski and Haiman. 
It also naturally yields the family of polynomials {G^ \ £ £ N} in the spirit of the original work by 
Lascoux, Leclerc and Thibon [8]. 
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